Meta Scheduling for Market-Oriented Grid and Utility Computing

Grid computing enables the sharing and aggregation of autonomous IT resources to deliver them as computing utilities to end users. The management of the Grid environment is a complex task as resources are geographically distributed, heterogeneous and autonomous in nature, and their users are self-interested. In utility-oriented Grids, users define their application requirements and compete to access the most efficient and cheapest resources. Traditional resource management systems and algorithms are based on system-centric approaches which do not take into account individual requirements and interests. To this end, market-oriented scheduling is an adequate way to solve the problem. But current market-oriented systems generally, either try to maximise one user’s utility or one provider’s utility. Such approaches fail to solve the problem of contention for cheap and efficient resources which may lead to unnecessary delays in job execution and underutilisation of resources. To address these problems, this thesis proposes a market-oriented meta-scheduler called “Meta-Broker”, which not only coordinates the resource demand but also allocates the best resources to users in terms of monetary and performance costs. The thesis results demonstrate that considerable cost reduction and throughput can be gained by adopting our proposed approach. The meta-broker has a semi-decentralised architecture, where only scheduling decisions are made by the meta-broker while job submission, execution and monitoring are delegated to user and provider middleware. This thesis also investigates market-oriented meta-scheduling algorithms which aim to maximise the utility of participants. The market-oriented algorithms consider Quality of Service (QoS) requirements of multiple users to map jobs against autonomous and heterogeneous resources. This thesis also presents a novel Grid Market Exchange architecture which provides the flexibility to users in choosing their own negotiation protocol for resource trading. The key research findings and contributions of this thesis are: • The consideration of QoS requirements of all users is necessary for maximising users’ utility and utilisation of resources. The uncoordinated scheduling of applications by personalised user-brokers leads to overloading of cheap and efficient resources. • It is important to exploit the heterogeneity between different resource sites/data centers while scheduling jobs to maximise the provider’s utility. This consideration not only reduce energy cost of computing infrastructure by 33% on average, but also enhance the efficiency of resources in terms of carbon emissions. • By considering both system metrics and market parameters, we can enable more effective scheduling which maximises the utility of both users and resource providers. This is to certify that (i) the thesis comprises only my original work, (ii) due acknowledgement has been made in the text to all other material used, (iii) the thesis is less than 100,000 words in length, exclusive of table, maps, bibliographies, appendices and footnotes.